1 link tagged with all of: agents + databricks + spark + data infrastructure
Click any tag below to further narrow down your results
Links
This a16z piece argues Databricks is quietly building the data infrastructure that agentic AI will need to run inside real companies, tracing its path from Spark to the lakehouse to Lakebase. It ties that infrastructure to a bigger claim: that front-line workers will regain decision-making power from the managerial hierarchy that dominated the 20th century.
- Databricks grew from Spark, an open-source engine Matei Zaharia wrote at Berkeley that kept data in memory and ran iterative workloads up to ten times faster than Hadoop.
- The "lakehouse" merged cheap data lake storage with warehouse reliability, and the company says its Lakehouse runs at a more than $1.5 billion revenue run-rate, growing over 100% year over year.
- Databricks bought Neon in 2025 to build Lakebase, which separates storage from compute and removes the pipelines between operational databases and analytics, a setup the author says agents will depend on.